The increased sophistication of computing, including artificial intelligence, data mining, statistics, machine learning, and database systems, requires a better paradigm for computer-based investigative analysis. The deficiencies of data mining—the use of statistical algorithms to extract patterns and insight from raw data—are an important catalyst for the new requirements for investigative analysis systems. With data mining, using computers to detect credit card fraud, to recommend the next movie to rent, or to find a good place to eat in a new city have become a part of our daily life.
However, data mining has limitations for use as an investigative analysis technique. From a technical perspective, automated data mining techniques are well-suited when the nature and composition of the underlying data does not change over time, the data is complete and clean, and the querier has some idea what he or she is looking for.
Unfortunately, many of hardest, most complex, and critical data problems that exist today do not have data characteristics that are well-suited for data mining techniques. These problems tend to involve data that comes from many disparate sources, is incomplete and inconsistent, and is created by those who are trying to avoid leaving a trail that is easy to follow. Further complicating matters, these problems are often bound up with social and privacy concerns. People generally are uncomfortable having a computer being a final arbiter when lives or livelihoods are on the line. Examples of where such problems can be found include fields such as intelligence analysis and police investigations. With these types of data problems, automated algorithmic approaches are sub-optimal because they omit human involvement at critical steps.
Accordingly, a need remains in the art for an investigative analysis system that addresses these and other issues.